1. Field of the Invention
The present invention relates generally to data input in a virtual environment and, more particularly, to inputting words and sentences by means of sensory gloves.
2. Description of the Related Art
A virtual environment is one where the user is placed entirely inside a computer-generated world. This is typically done by means of various types of communication gear attached or strapped to the user""s bodyxe2x80x94for example, goggles to create a visual tableau, earphones to provide an audible counterpart, and other instruments that generate sensory input for the user. In order to immerse the user in a virtual world, this gear serves two functions: one is to provide sensory input to the user, the other is to provide the virtual world system with information about the user. For instance, headgear will tell the system the direction of the user""s view and sensory gloves will tell the relative locations of the hands and fingers of the user.
In the virtual environment, traditional means of inputting data to the system are neither available nor practical. Because a keyboard cannot be easily attached to the human body, the user usually has to leave the virtual environment in order to enter text data. This inability to enter key data such as text in a virtual environment will become more problematic over time, as the potential of virtual world representations of complex file systems, flowcharts, architectural drawings, and event simulations are realized.
Therefore, other means of text input need to be created for virtual environments. While there are many possible solutions to this problem, including speech recognition, some type of hand or finger manipulation seems to be the most intuitive way to enter text.
Sensory gloves have been considered as appropriate instruments of such text entry. Grimes, U.S. Pat. No. 4,414,537, discloses a glove with sensors positioned to detect joint angles and positions of each finger and further discloses using such a glove to replace a keyboard. However, Grimes"" user must learn a new set of static gestures to specify letters. Although Grimes discloses entering data into a virtual world system using sensory gloves, the user must learn new skills that are not intuitive or obvious.
Zimmerman et al., U.S. Pat. No. 4,988,981, discloses sensory gloves that could be used to manipulate objects in a virtual world, such as moving a cursor or performing operations based on gesture and position of the user""s hands. The virtual object to be manipulated in Zimmerman et al. could be a representation of an input device such as a keyboard.
There are severe limitations on the prior art use of sensory gloves for key input. A first problem involves the ability to recognize when a simulated finger press is being made. Sensory gloves are typically used to recognize only coarse hand gestures, such as select, grasp, and drop, not the fine gestures needed to differentiate individual finger movements. When fingers are used to represent individual keys, low-cost low-resolution sensory gloves present several problems. Correlated finger movements called sympathetic finger movements, which occur when one finger is pressed down to input a key and the neighboring finger moves a certain distance along with it, create confusing data. For example, FIG. 2A illustrates what happens as a person simulates a press of the pinkie finger. As the pinkie finger presses down (indicated by the peak labeled A), the ring finger (line B) parallels the movement. If the ring finger has a lower threshold (value at which the system determines there has been a simulated press) than the pinkie, the system may determine that the ring finger was the one being pressed.
The phenomenon of finger amplitude variation is also a problem. Two different users may use differing amounts of movement to indicate a finger press. In addition, over time a single user varies the amount of movement to indicate a press. For example, FIG. 2B shows the same intended pinkie press as FIG. 2A, but made ten minutes later. Although the user started by using a right-hand index finger press value of 130 in FIG. 2A, the user is only using a value of 55 when making the same finger press in FIG. 2B.
Lastly, the measurement of finger movements is inherently noisy, which causes false peaks. More exactly, sympathetic finger motions, minor motions caused by shaking hands, and glove output noise produce jagged curves in the finger motion data, as shown in FIG. 2C. These jagged curves can lead the system to both missing presses and recognizing unintended ones.
A second problem is caused by the fact we are limited to ten fingers, but we use a 26-letter alphabet with punctuation symbols. If every finger of the sensory gloves is used to represent an individual key, it would not be possible to represent an entire alphanumeric keyboard by single key entry. It would be possible to use combinations of individual keystrokes to represent each alphanumeric character, but such a procedure would be burdensome.
One solution to this problem is to use an overloaded keyboard, whereby each key, or finger, would represent several different alphanumeric characters. Such a keyboard layout is shown in FIG. 1, where multiple letters are mapped to certain fingers. The left-hand ring finger, for example, could represent xe2x80x9csxe2x80x9d, xe2x80x9cvxe2x80x9d, and xe2x80x9cxxe2x80x9d, the right-hand index finger could represent xe2x80x9cjxe2x80x9d, xe2x80x9cu,xe2x80x9d xe2x80x9cyxe2x80x9d, xe2x80x9chxe2x80x9d and xe2x80x9cnxe2x80x9d, and so on for each finger. This presents a new difficulty: how will the virtual system be able to tell which alphanumeric character was intended, out of the multiple characters that one key input represents?
This character ambiguity can be resolved by exploiting the statistical and grammatical constraints of language in order to reconstruct the text. This process of disambiguation uses probabilistic analysis and a dictionary. There have been various approaches to this overloaded keyboard problem in the field of text entry. One method decodes the text word-by-word using a weighted tree structure called a xe2x80x9cProbabilistic History Treexe2x80x9d (See, Litvak and Shamir, Efficient algorithms for constructing and employing variable-length Markov models of language, Technical Report, Dept. of Computer Science, Tel-Aviv University, Israel, 1996). Another method, which has also been applied to handwriting recognition, decodes the text sentence-by-sentence using a hash table (See, H. Rau and S. Skiena, Dialing for documents: An experiment in information theory, Journal of Visual Languages and Computing, pages 79-95, 1996). But all of these methods were directed towards the overloaded alphabetic keyboard of a telephone keypad.
Therefore, a need exists for a method and system that can reconstruct text from ambiguous characters, as well as reliably recognize the fine movements that represent presses using low-cost, low-resolution sensory gloves.
It is therefore an object of the present invention to provide a system and method of recognizing fine finger movements, such as key entry on a virtual keyboard, using low-cost, low-resolution sensory gloves. In particular, the system and method solves the problem of sympathetic finger movements by examining the relative motions of neighboring fingers. The system and method also solves the problem of finger amplitude variation by performing an initial calibration, providing the user with visual and aural feedback, and using dynamic thresholds to evaluate the finger movements. Lastly, the system and method eliminates the inherent noise in the finger movement data by using a low-pass Gaussian filter.
It is another object of the present invention to provide a system and method of resolving ambiguous characters entered on an overloaded keyboard. This ambiguous character resolution, or reconstruction, is performed by recognizing blank characters, thus separating out words as key sequences; matching each key sequence with one or more words in a dictionary; constructing or partially matching the words with no dictionary match; generating all possible partial sentences from the possible words; and then determining the most likely partial sentence by assigning probabilities to each possible partial sentence.
The present invention builds on the idea that sensory gloves can be used to create a virtual keyboard, where finger movements can be translated into key entry by a virtual world system. In the exemplary embodiment, the user does not actually see a simulated keyboard in the video goggles, but is shown which fingers represent which letters and when the system recognizes a finger press as text input. In the following text, the term xe2x80x9cpressxe2x80x9d is used to describe a finger motion that is mimicking a finger press on a keyboard, and is intended by the user to indicate that the key represented by that finger has been pressed. The terms xe2x80x9ckeyboard layoutxe2x80x9d refers to how the alphanumeric characters are assigned to each finger and xe2x80x9ckeyboardxe2x80x9d to the system by which the sensory gloves are manipulated to represent alphanumeric characters.